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ABSTRACT 


Early  Synthetic  Prototyping  (ESP)  is  a  new  concept  in  which  capability  and 
material  developers  use  an  online  game  to  crowdsource  ideas  from  online  players  in  order 
to  increase  viable  synthetic  prototypes.  In  entertainment  games,  players  often  create 
videos  of  their  game  play  to  share  with  other  players  to  demonstrate  how  to  complete  a 
segment  of  a  game.  This  thesis  explores  similar  self-recorded  videos  of  ESP  game  play 
and  determines  if  they  provide  useful  data  to  capability  and  material  developers  that  can 
influence  the  early  design  process,  or  if  the  videos  affect  the  ESP  process  itself. 

The  study  shows  that  user  videos  affect  player  behavior  as  well  as  increase 
engagement  and  entertainment  for  the  players,  which  serves  to  maintain  a  large  player 
population  essential  to  ESP  success.  The  exact  reasons  for  increased  engagement  and 
entertainment  are  unclear  and  are  topics  for  further  investigation.  These  results  are 
important  to  ESP  developers  because  if  ESP  game  developers  can  increase  the 
engagement  and  fun  in  playing  their  games,  it  will  increase  participation  and  willingness 
to  contribute  ideas  and  strategies  with  other  players.  The  increase  in  contributions  and 
participation  will  then  lead  to  an  increase  in  the  number  of  early  prototypes  that  can  be 
analyzed  and  potentially  used. 
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EXECUTIVE  SUMMARY 


For  the  first  time  in  40  years,  the  Army  has  updated  its  operating  concept,  which 
is  a  shift  from  how  to  “fight,  outnumber,  and  win”  to  how  to  “win  in  a  complex  world” 
(Perkins,  2014).  In  order  to  win  in  a  complex  world,  the  Army  will  need  to  not  only 
anticipate  future  threats  but  also  innovate  in  order  to  ensure  its  forces  are  able  to  bring 
overmatching  capabilities  to  bear  on  it  enemies.  (U.S.  Army  Training  and  Doctrine 
Command  [TRADOC],  2014).  One  way  the  Army  seeks  to  improve  and  speed  up  its 
innovation  is  with  Early  Synthetic  Prototyping  (ESP). 

ESP  is  a  new  concept  in  which  capability  and  material  developers  use  an  online 
game  to  crowdsource  ideas  from  online  players  in  order  to  increase  the  viability  of 
synthetic  prototypes.  One  current  trend  in  online  game  communities  is  the  creation  and 
sharing  of  self-recorded  game  play  videos.  Since  ESP  will  be  an  online  game,  users  may 
create  self-recorded  videos  of  their  game  play  for  other  users  to  watch.  This  thesis  seeks 
to  answer  the  following  questions  with  regard  to  user  created  videos  in  an  ESP  game 
community: 

1.  Is  it  reasonable  to  expect  ESP  game  players  to  create  and  narrate  game 
play  video  after  action  report  (AAR)s? 

2.  Does  availability  of  game  play  video  AARs  increase  user  feedback  and 
participation  in  ESP  game  play? 

3.  Will  game  play  video  AARs  provide  useful  feedback  to  developers? 

4.  Will  game  play  video  AARs  provide  useful  information  to  other  players 
and  enhance  the  game  play? 

In  order  to  answer  these  questions,  the  author  conducted  a  human-in-the-loop 
study  with  two  groups  playing  an  ESP-style  military  game.  The  first  group  conducted  a 
mission  and  then  recorded  a  game  play  video.  The  second  group  watched  the  videos,  and 
then  played  the  game. 

Results  of  the  study  show  that  the  user  videos  affect  the  players’  behavior  as  well 
as  created  a  more  entertaining  environment  for  the  players  than  one  without  videos.  The 
exact  reasons  why  the  videos  increased  the  fun  platers  had  within  the  environment  are 


XV 


unknown  and  future  studies  should  further  investigate  the  cause.  These  results  are 
important  to  ESP  developers  because  if  ESP  game  developers  can  increase  the  fun  in 
their  games,  it  should  increase  overall  participation  and  buy-in  of  their  players.  (Vogt, 
Megiveron,  &  Smith,  2015).  The  increase  in  buy-in  and  participation  should  lead  to 
more  opportunities  for  developing  and  testing. 
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I. 


INTRODUCTION 


Since  the  early  1980s  and  prior  to  the  newly  published  Army  Operating  Concept 
(AOC),  the  United  States  Army  used  the  Airland  Battle  Doctrine  as  a  blueprint  for  how  it 
would  equip  an  Army  designed  to  fit  a  known  enemy  on  a  known  terrain.  (Perkins  2014). 
Over  the  past  decade,  the  operational  environment  that  the  Army  fights  in  has  changed 
and  thus,  the  operating  concept  has  changed  as  well.  According  to  the  new  AOC,  “the 
environment  the  Army  will  operate  in  is  unknown.  The  enemy  is  unknown,  the  location  is 
unknown,  and  the  coalitions  involved  are  unknown”  (TRADOC,  2014,  p.  iii).  The  title  of 
the  new  AOC  and  the  problem  the  Army  is  now  faced  with  solving  is  how  to  “Win  in  a 
Complex  World”  (TRADOC,  2014). 

In  the  Army’s  newly  published  AOC,  “complexity  is  defined  as  an  environment 
that  is  not  only  unknown,  but  unknowable  and  constantly  changing.  The  Army  cannot 
predict  who  it  will  fight,  where  it  will  fight,  and  with  what  coalition  it  will  fight” 
(TRADOC,  2014,  p.  iii).  This  new  operating  concept  will  require  the  institutional  Army 
to  be  more  adaptive  and  innovative  in  its  mission  to  equip  and  train  the  operational  Army 
“to  ensure  that  Army  forces  are  manned,  trained,  and  equipped  to  overmatch  enemies  in 
order  to  seize,  retain,  and  exploit  the  initiative”  (p.  iv). 

The  Army  defines  “overmatch”  as  the  “application  of  capabilities  or  use  of  tactics 
in  a  way  that  renders  an  adversary  unable  to  respond  effectively”  (TRADOC,  2014,  p.  9). 
As  technology  proliferation  increases,  the  adversarial  forces  that  the  Army  will  face  can 
diminish  the  Army’s  technological  advantage  in  two  ways.  They  can  either  use  new 
technologies  to  increase  their  own  or  mitigate  the  Army’s  capabilities.  This  could  in  turn 
then  lead  to  either  an  enemy  overmatch  or  the  loss  of  the  Army’s  overmatch.  The  Army 
aims  to  prevent  this  loss  of  overmatch  by  rapidly  developing  new  capabilities.  Therefore, 
as  the  need  to  speed  up  the  development  of  new  capabilities  to  field  to  the  operating  force 
becomes  increasingly  important,  the  institutional  Army  seeks  to  use  innovation  to  solve 
this  problem  (TRADOC,  2014). 
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Innovation  is  a  key  tenet  of  the  Army  and  the  one  that  it  seeks  to  use  to  maintain 
overmatch  against  existing  and  potential  adversaries.  TRADOC  describes  innovation  in 
the  Army  Operation  Concept  as 

the  result  of  critical  and  creative  thinking  and  the  conversion  of  new  ideas 
into  valued  outcomes.  Innovation  drives  the  development  of  new  tools  or 
methods  that  permit  Army  forces  to  anticipate  future  demands,  stay  ahead 
of  determined  enemies,  and  accomplish  the  mission.  Innovation  is 
particularly  important  in  organizations  that  develop  capabilities  as  well  as 
those  that  train,  equip,  and  sustain  forces.  (TRADOC,  2014,  p.  22) 

From  this  critical  thinking,  the  Army  determines  the  needed  capabilities  to  equip 
the  future  forces.  Such  thinking  also  provides  the  answers  to  how  tomorrow’s  leaders  will 
employ  future  and  existing  capabilities  to  accomplish  a  multitude  of  missions  in  various 
environments  (TRADOC,  2014).  Formally,  this  critical  thinking  supports  the  Army’s 
Warfighting  Challenge  No.  4: 

How  to  maintain  an  agile  institutional  Army  that  ensures  combat 
effectiveness  of  the  total  force,  supports  other  services,  fulfills  DOD 
[Department  of  Defense]  and  other  agencies’  requirements,  ensures 
quality  of  life  for  Soldiers  and  families,  and  possesses  the  capability  to 
surge  (mobilize)  or  expand  (strategic  reserve)  the  active  Army. 
(TRADOC,  2014,  p.  32). 

Currently,  the  Army  uses  an  acquisition  process  in  which  the  capability  developer 
conducts  a  needs  analysis  to  determine  if  any  gaps  exist  in  the  current  and  future  forces 
capabilities  needed  to  accomplish  their  required  missions  (TRADOC,  2014).  If  any  gaps 
exist,  the  capability  developers  identify  them  and  develop  the  requirements  necessary  to 
fill  them.  Then,  they  give  these  requirements  to  material  developers  to  build  the  system 
and  field  it  to  the  operational  Army.  This  process  utilizes  very  few  capability  developers 
and  produces  very  few  prototypes  for  testing.  The  Army  could  improve  this  process  by 
increasing  the  number  of  critical  thinkers  participating  in  the  design  process,  which  in 
turn  should  also  increase  the  number  of  viable  prototypes  for  future  testing.  One  way  the 
Army  seeks  to  leverage  innovation  and  improve  the  acquisition  process  in  this  manner  is 
to  use  Early  Synthetic  Prototyping.  (ESP). 
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ESP  is  a  new  initiative  from  the  Army  Capabilities  Integration  Center  (ARCIC) 
that  seeks  increase  the  amount  of  critical  thinkers  participating  in  the  design  of  future 
capabilities  with  the  use  of  an  online  military  themed  game.  This  game  will  allow  users 
outside  of  the  traditional  capability  and  material  development  community  to  participate  in 
the  design  process.  ESP  will  allow  Soldiers  from  across  the  Army  to  test  and  evaluate 
synthetic  prototypes  and  then  provide  feedback  to  developers  (Vogt,  2016).  The  Army 
will  leverage  innovation  by  crowdsourcing  ideas  and  evaluations  from  thousands  of 
Soldiers  to  ensure  the  Army  of  the  Euture  maintains  overmatch  against  its  potential  foes. 
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II.  BACKGROUND 


The  Army  is  looking  to  ESP  to  not  only  increase  the  number  of  viable  prototypes, 
but  also  produce  feedback  from  the  actual  users  early  in  the  design  process.  The  main 
vehicle  for  this  feedback  will  be  the  use  of  an  online  game  and  an  online  community. 
Therefore,  the  focus  of  this  thesis  concerns  how  to  best  facilitate  feedback  from  players 
in  a  form  useful  to  materiel  developers.  One  of  the  most  recent  trends  in  online  gaming  is 
the  creation  and  sharing  of  ideas  using  self-recorded  game  videos.  Players  of 
entertainment  games  often  create  videos  of  their  game  play  to  demonstrate  how  they 
solved  a  problem  or  completed  a  mission.  These  videos  are  then  posted  on  public 
websites  for  sharing.  If  players  are  willing  to  create  and  share  videos  for  commercial 
entertainment  games,  would  ESP  players  similarly  create  videos  in  an  ESP  game 
environment? 

A.  EARLY  SYNTHETIC  PROTOTYPING 

According  to  ARCIC, 

initial  research  indicates  that  computer  based  gaming  environments  can 
provide  a  cost-effective  and  engaging  means  to  enable  Soldiers  and 
leaders  across  the  Army  to  explore  future  capabilities,  develop  solutions  to 
associated  challenges,  provide  related  recommendations,  and  collaborate 
with  one  another  to  inform  concept  and  capability  developers.  (Vogt, 

2016,  p.  1) 

I.  What  Is  ESP? 

ESP  is  a  new  concept  the  United  States  Army  is  currently  exploring.  The  two 
primary  agencies  conducting  the  research  are  ARCIC  and  the  United  States  Assistant 
Secretary  of  the  Army  for  Acquisition,  Eogistics,  and  Technology  (ASA[AET]) 
(Richmond,  n.d.).  According  to  Vogt  (2016),  the  Early  Synthetic  Prototyping  Project 
Eead  for  ARCIC,  the  Army  has  divided  the  development  of  ESP  into  four  areas  of  effort: 
collaboration,  feedback  and  game  analytics,  the  game  environment,  distribution  and 
network  capability,  and  research. 
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The  largest  effort  in  the  development  of  ESP  is  the  development  of  collaborative 
environment.  The  collaborative  environment  will  provide  “the  means  for  Soldiers  to 
provide  explicit/qualitative  feedback  through  surveys,  polls,  discussion  boards  as  well  as 
implicit/quantitative  feedback  through  automated  game  analytic  tools  to  identify  trends  in 
game  play”  (Vogt,  2016,  p.  2).  The  next  effort  is  the  development  of  the  game 
environment.  Many  military-based  games  are  available  within  the  Department  of  Defense 
(DOD)  and  as  commercial-off-the-shelf  solutions;  however,  since  the  goal  of  an  ESP 
game  is  much  different,  most,  if  not  all,  of  the  other  solutions  available  will  not  be 
suitable  for  an  ESP  game  (2016).  An  ESP  game  will  need  to  collect  data  unique  to  the 
acquisition  and  material  development  process.  This  will  require  the  Army  to  build  two 
types  of  games.  The  first  game  will  be  a  first  person  shooter  game.  It  will  evaluate  how 
Soldiers  use  and  interact  with  individual  systems.  The  second  game  will  be  a  strategy 
game  played  at  a  higher  echelon  to  evaluate  organization  and  employment  of  future 
systems  (2016).  The  third  effort  for  the  development  of  ESP  is  distribution  and  network 
capability.  This  effort  focuses  on  the  need  “to  deliver  and  host  the  game  environment(s) 
and  the  feedback,  collaboration  and  game  analytic  tools”  (p.  2).  Einally,  the  last  effort  is 
research.  Since  ESP  is  a  new  concept,  the  Army  will  need  to  collaborate  with  both 
industry  and  academic  institutions  to  ensure  that  the  body  of  knowledge  with  regard  to 
ESP  is  continuously  growing  (2016). 

ESP  essentially  seeks  to  improve  not  only  the  Army’s  acquisition  process  but  also 
doctrinal  and  organizational  development  processes  by  making  two  major  changes  to  the 
current  acquisition  process.  Eirst,  it  seeks  to  increase  the  number  of  viable  design  points 
by  using  crowdsourcing  to  develop  and  evaluate  virtual  prototypes.  Second,  it  aims  to  put 
the  soldier  in  the  middle  of  the  design  process.  In  its  final  state 

ESP  is  envisioned  to  provide  a  synthetic  game  environment  where 
engineers  and  concept  developers  can  rapidly  model  multiple  potential 
capabilities/prototypes.  Soldiers,  on  and  off  duty,  are  able  engage  in 
various  scenarios  specifically  designed  to  elicit  feedback  that  address 
research  questions  associated  with  the  Army  Warfighting  Challenges  as  it 
relates  to  doctrine,  organization,  and  materiel  solutions.  (Vogt,  2016,  p.  1) 
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Currently,  the  Army’s  system  for  material  development  does  not  inelude  the 
Solider  until  the  testing  phase  of  new  equipment.  This  system  allows  for  very  little 
collaboration  between  all  stakeholders  involved  in  the  acquisition  process.  Using  ESP  for 
development,  ESP  brings  the  Soldier  to  the  center  of  the  process  and  collaboration 
between  all  agencies  is  possible  not  only  through  the  ESP  collaboration  environment,  but 
also  from  the  game  metrics  specifically  designed  to  inform  developers  on  the  evaluation 
of  proposed  systems  (see  Eigure  1). 


Eigure  1.  Current  Development  System  versus  the  ESP  System.  Source: 

Vogt,  Darken,  McGroarty,  Smith,  Perriello,  and  Phillips  (2014). 
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Eigure  2.  Cost  of  a  Design  Change.  Source:  Vogt  et  al.  (2014). 


o 

O) 

c 

(U 

£ 

u 

c 

O) 

'ifl 

V 

O 

(D 


Ul 

O 

u 


Early  Synthetic 
Prototyping 

Enables  early 
feedback  to  drive 
design  when  changes 
are  ij^xpensive 


Current  method  does  not 
consistently  get  Soldier 
input/feedback  until 
production/fielding  when  design 
changes  are  costly  or  prohibitive 


Basic 

Concept 

Evaluation 


Detailed  Design  Real  Prototype 

□^1 


Live 

Testing 


Enabinfl 

S«.T 


v-acquisitioi 
Concepts, 
^rimentation] 
i  Prototyping] 


i 

!L 


Motcriol 

Sofuticn 

Analysis 


Technology 

Oevelopment 


BV 


Prodluction 

Li’^l 


Fielding 


Engineering  artd 
Manutacturing 
Doveloprrtont 

—  — 


W  P  i  oc''  ^toc 


Production  Operations 
and  and 

Deployment  Support 


7 


Also,  the  current  design  method  used  by  the  Army  does  not  gather  feedback  from 
actual  users  until  late  in  the  development  timeline,  when  actual  prototypes  are  available 
for  testing  (see  Figure  2).  This  is  expensive  and  limiting.  ESP  seeks  to  enable  an 
environment  in  which  soldiers  can  provide  feedback  based  on  virtual  prototypes.  Since 
these  virtual  prototypes  are  available  much  earlier  in  the  development  timeline,  the 
changes  are  much  cheaper.  ESP  seeks  to  bring  “Soldiers,  scientist,  engineers  and  concept 
developers  into  a  common  environment  to  rapidly  assess,  modify,  and  experiment  with 
future  equipment — without  bending  a  single  piece  of  metal.  The  core  objective  of  ESP  is 
to  transition  ideas  and  concepts  from  a  good  idea  on  a  dry  erase  board  to  the  game 
environment  that  thousands  of  soldiers  across  the  Army,  on  and  off  duty  are  able  to 
assess.  This  assessment  will  simultaneously  inform  science  technology  investments  as 
well  as  inform  doctrine,  organization  and  training/education  development”  (Vogt, 
2013,  p.  1). 

2.  Previous  ESP  Projects 

The  Office  of  Secretary  of  Defense  (OSD)  Engineered  Resilient  Systems 
sponsored  the  first  ESP  project  in  December  2013.  This  pilot  study’s  objective  was  to 
examine  the  feasibility  of  using  gaming  a  as  a  means  to  evaluate  future  capabilities.  This 
study  resulted  in  a  thesis  experiment  conducted  at  the  Naval  Postgraduate  School  (Vogt, 
2016).  The  study  sought  to  answer  the  following  questions:  “What  feedback  can  be 
gathered  from  [ESP]  game  play”  and  “Would  that  feedback  be  valuable”  (Murray,  2014, 
p.  xix).  The  results  of  the  study  concluded  that  information  collected  from  game  play  in 
an  ESP  environment  is  both  relevant  and  useful  (Murray,  2014). 

In  December  of  2014,  the  Army  Modeling  and  Simulation  Office  (AMSO) 
sponsored  another  ESP  study.  In  this  study  conducted  by  ARCIC,  the  researchers  focused 
on  two  major  questions.  Eirst,  they  wanted  to  find  out  what  would  motivate  soldiers  to 
play  ESP  games  in  their  off  time.  Second,  they  sought  to  determine  if  any  of  the  feedback 
and  metrics  collected  from  their  pilot  study  were  useful  to  capability  and  material 
developers.  Results  of  the  study  indicated  that  a  large  majority  of  soldiers  wanted  to 
participate  and  play  ESP  games  if  they  knew  that  the  Army  was  using  their  feedback  to 
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develop  future  weapon  systems.  Furthermore,  the  researchers  also  felt  that  the  quality  of 
the  feedback  received,  although  through  surveys  and  discussions,  was  valuable  (Vogt  et 
ah,  2015). 

B.  GAMING  COMMUNITIES  AND  THE  “LET’S  PLAY”  PHENOMENON 

Video  games  are  more  popular  now  than  ever  and  that  means  big  business  to  the 
game  industry.  Over  155  million  Americans  play  video  games,  more  than  half  of  the 
households  in  America  have  a  dedicated  game  system,  and  total  consumer  spending  on 
video  games  in  the  U.S.  exceeded  20  Billion  dollars  in  2015  (Entertainment  Software 
Association  [ESA],  2015).  Gaming  numbers  may  be  even  higher  for  Soldiers.  One  recent 
study  suggested  that  over  76  percent  of  the  Army’s  Soldiers  who  would  participate  in 
ESP  games  already  play  video  games.  Over  70  percent  of  these  soldiers  play  at  least  two 
hours  a  week  and  over  50  percent  of  them  play  over  ten  hours  (Vogt  et  al.,  2015). 
Compare  that  with  recent  statistic  from  the  ESA  that  only  42  percent  of  Americans  play 
video  games  more  than  three  hours  per  week  (ESA,  2015).  These  statistics  suggest  not 
only  that  the  average  Soldier  plays  more  video  games  than  the  average  American,  but 
also  that  the  Army  has  a  prime  environment  in  which  to  create  a  community  of  ESP 
players. 

Creation  and  management  of  a  gaming  community  in  which  players  will  be  able 
to  connect  with  other  players  and  developers  are  paramount  to  the  success  of  the  ESP 
program.  These  communities  will  be  the  environment  in  which  players  will  both  share 
their  experiences  and  provide  feedback.  These  communities  will  also  allow  developers  to 
both  actively  and  passively  collect  the  desired  metrics  on  the  various  prototypes  tested. 
(Vogt,  2016) 

While  the  ESP  developers  will  seek  to  create  and  manage  this  online  community, 
history  has  shown  that  most  likely  this  community  will  take  on  a  personality  of  its  own. 
The  emergence  of  gamming  channels  on  YouTube  is  one  such  an  example  of  players 
seeking  an  area  to  collaborate  as  well  as  watch  other,  more  skillful  players  (lannitti, 
2015).  While  this  may  sound  foreign  to  people  who  do  not  play  games  or  identify  as 
gamers,  the  numbers  show  that  this  phenomenon  is  becoming  more  and  more 
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mainstream.  The  number  one  ehannel  on  YouTube  in  both  subseribers  and  views  (see 
Figure  3)  is  a  25-year  -old  gamer  from  Sweden  named  Felix  Kjellberg,  but  goes  by  the 
sereen  name  of  PewDiePie  (MeConnell,  2014).  Not  only  is  Kjellberg  No.  1  in  terms  of 
subseribers  and  video  views,  he  is  the  No.  1  video  ereator  on  YouTube,  earning  over  $12 
million  in  2015  alone  (Berg,  2015). 


Figure  3.  Top  YouTube  Video  Views  and  Subseriber  Channels.  Adapted 

from  SoeialBlade  (2016). 


So,  why  do  gamers  floek  to  watch  other  gamers  play  video  games?  According  to 
Nick  lannitti.  Digital  and  Social  Media  Strategist  at  Fuel  Entertainment,  this  behavior  is 
nothing  we  have  not  seen  before.  lannitti  says  that  the  gamers  that  are  creating  videos  are 
“simply  following  the  preexisting  laws  of  nature  in  something  called  Emergent  Behavior. 
We  can  observe  emergent  behavior  in  everything  from  ant  colonies  to  the  largest  of 
cities.  The  premise  of  emergent  behavior  is  that  we  are  all  connected  through  networks 
(both  online  &  offline)  and  that  we  naturally  self-organize  across  our  networks  to  form 
higher  levels  of  order”  (lannitti,  2015,  para.  7).  In  essence,  gamers  are  looking  for  other 
gamers  who  share  their  interests.  Thanks  to  advances  in  technology  and  the  emergence  of 
the  Internet  and  YouTube,  these  communities  can  now  connect  anywhere,  at  any  time. 
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What  exactly  is  a  YouTube  game  channel  like  “Let’s  Play”  (LP)?  Essentially,  LP 
is  an  online  forum  where  “folks  play  through  a  game,  continually  document  their 
progress  via  photos  or  video,  and  add  their  own  commentary  to  complete  the  package. 
This  content  is  typically  posted  in  message  board  threads,  which  draw  in  an  enthusiastic 
audience  that’s  eager  to  discuss  the  game,  contribute  suggestions,  and  even  concoct  their 
own  creative  works  directly  related  to  the  title  in  question.  The  result:  a  highly  interactive 
community  that’s  created  hundreds  upon  hundreds  of  hours  of  entertainment”  (Mackey, 
2011,  para.  2).  Mackey’s  description  of  LP  looks  quite  similar  to  ETC  Vogt’s  description 
of  what  an  ESP  game  will  provide.  Vogt  states  that  an  ESP  community  will  be  “the 
means  for  Soldiers  to  provide  explicit/qualitative  feedback  through  surveys,  polls, 
discussion  boards  as  well  as  implicit/quantitative  feedback  through  automated  game 
analytic  tools  to  identify  trends  in  game  play”  (Vogt,  2016,  p.  2).  It  makes  sense  that  ESP 
developers  need  to  be  concerned  that  their  players  may  seek  other  forums  for 
collaboration,  especially  if  the  ESP  developers  can  produce  an  ESP  game  that  is  near  the 
fun  level  of  other  commercial  games  that  ESP  developers  will  be  competing  with  for 
Soldiers’  time.  The  issue  of  ESP  game  players  collaborating  outside  of  the  ESP 
community  could  mean  a  loss  of  valuable  game  metrics  if  the  information  that  users  share 
in  these  types  of  forums  is  useful  to  both  developers. 
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III.  METHODOLOGY,  RESULTS  AND  DISCUSSION 


A  human-in-the-loop  experiment  which  was  approved  the  NPS  Institutional 
Review  Board,  protocol  number  NPS20160038.0  was  conducted  at  the  Naval 
Postgraduate  School  (NPS)  Modeling,  Virtual  Environments  and  Simulations  (MOVES) 
Institute.  The  experiment  was  a  case  study  to  determine  how  effective  self-recorded  AAR 
videos  of  users  playing  ESP  games  are  in  the  collection  of  feedback  and  the  usefulness  of 
the  information  gained  to  developers.  The  experiment  utilized  the  Army’s  Virtual 
Battlespace  3  (VBS3)  game  to  simulate  an  ESP-like  game.  VBS3  is  a  simulation  training 
solution  develop  by  Bohemia  Interactive  and  used  by  the  United  States  Army  for 
collective  training  and  mission  rehearsals  (Bohemia  Interactive  Simulations,  n.d.). 
Researchers  chose  VBS3  as  the  software  for  this  experiment  to  simulate  an  ESP  game 
because  it  was  readily  available  and  flexible  enough  to  create  an  adequate  scenario  for 
video  creation  and  narration. 

A.  SCOPE 

The  scope  of  this  thesis  will  be  limited  to  analyzing  the  use  and  effects  of  video 
AARs  in  ESP  game  play  systems.  Information  derived  from  ESP  games  for  design  and 
development  purposes  can  run  the  gamut,  and  therefore  other  collection  techniques  will 
not  be  considered.  Also,  creation  and  management  of  the  online  game  community  itself 
would  be  a  very  difficult  task;  therefore,  this  thesis  will  assume  that  of  such  a  community 
already  exists.  A  surrogate  will  be  provided  locally. 

B.  EXPERIMENT  DESIGN 

The  experiment  utilized  two  groups  of  participants.  Researchers  recruited 
participants  for  both  groups  and  randomly  assigned  participants  to  either  to  Group  1  or 
Group  2. 

I.  Participants 

The  experiment  required  40  participants  and  took  place  over  10  sessions  with  four 
participants  per  session.  Researchers  solicited  participants  via  flyer’s  distributed  in 
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person  and  via  the  NPS  muster  page  daily  announcements.  The  target  audience  for  the 
experiment  was  NPS  students  with  military  experience  who  regularly  play  video  games. 
However,  due  to  volunteer  turn  out,  experimenters  accepted  participation  from  all 
respondents. 

Participants  formed  two  teams,  either  red  or  blue.  For  each  session,  during  the 
initial  in  brief  and  introduction,  the  researcher  queried  all  four  participants  to  determine 
which  players  had  the  most  video  game  experience.  Based  on  that  knowledge,  the 
experimenter  divided  the  participants  to  balance  the  teams  with  regard  to  game 
experience.  However,  it  should  be  noted  that  team  balance  is  not  a  key  issue  here  since 
“performance”  itself  was  not  measured.  Our  interest  here  was  in  behavior  and  strategy 
employed,  not  in  how  well  the  team  played  the  scenario. 

The  target  audience  for  this  study  was  military  gamers;  however,  a  few  factors 
limited  the  ability  to  ensure  that  each  participant  was  the  ideal  volunteer.  First,  the  time 
constraints  forced  researchers  to  conduct  the  study  over  a  three- week  period.  Second,  this 
study  required  each  session  to  have  four  volunteers.  These  two  requirements  made  it 
difficult  to  find  enough  target  volunteers  with  similar  schedule  availability  to  conduct  the 
experiments.  See  Table  1  for  participant  demographics. 
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Table  1.  Participant  Demographics 
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2.  Scenario  Design 

The  scenario  for  each  session  was  the  same.  The  red  team  occupied  an  abandoned 
prison  and  held  an  American  journalist  hostage.  The  red  team’s  mission  was  to  protect 
the  hostage  and  defend  their  position  until  a  red  quick  reaction  force  (QRF)  could  arrive. 
The  red  QRF  would  arrive  8-10  minutes  after  either  the  initial  engagement  or  when  red 
forces  spotted  blue  forces.  The  blue  team’s  mission  was  to  move  from  their  release  point 
to  the  prison,  secure  the  prisoner,  and  then  move  to  the  pick  zone  (PZ)  for  extraction. 

The  red  team  consisted  of  four  opposing  forces  (OPFOR)  soldiers.  Two  red 
participants  played  two  of  the  four  soldiers,  and  the  game’s  artificial  intelligence  (AI) 
controlled  the  other  two.  The  red  team  could  place  the  two  AI  soldiers  in  either  the 
towers  to  provide  overwatch  of  the  prison  or  as  roving  guards  to  patrol  the  perimeter  of 
the  prison  complex.  The  red  team  would  win  if  they  either  defeated  the  blue  force  attack 
or  prevented  the  blue  forces  from  securing  the  prisoner  until  their  QRF  arrived. 

The  blue  team  consisted  of  four  U.S.  Army  Soldiers.  Two  blue  participants  played 


two  of  the  four  soldiers  and  the  game’s  AI  controlled  the  other  two.  The  blue  force 
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commander,  however,  did  have  the  ability  to  control  the  AI  soldiers  by  using  the 
commands  within  the  game.  The  blue  team  won  the  game  if  they  successfully  secured  the 
prisoner  and  reached  the  PZ  before  red  QRF  forces  arrived. 

3.  Group  1  Experiment 

For  Group  1,  participants  began  by  completing  a  few  tutorials  and  a  practice 
scenario.  The  tutorials  taught  the  users  basic  movements  and  game  functions  needed  for 
the  game  scenario.  After  the  tutorials,  users  conducted  a  rehearsal  scenario  where  they 
had  the  opportunity  to  practice  the  execution  of  the  various  controls  needed  to  complete 
the  mission.  After  the  rehearsal  scenario,  the  moderator  gave  each  team  a  mission  brief. 

After  this  brief,  each  team  had  5  minutes  to  plan  their  mission.  The  blue  team  had 
to  choose  one  of  three  possible  release  points  where  they  would  begin  their  mission.  The 
red  team  had  to  choose  one  of  two  defense  strategies,  either  roving  guards  or  towers. 

Once  the  game  was  complete,  the  experiment  moderators  assisted  each  user  in  the 
creation  of  an  AAR  video.  This  video  used  actual  game  play  footage  and  a  planning  map 
overview  for  the  users  to  narrate  and  analyze  their  execution  of  the  mission  and  the 
decisions  they  made. 

The  following  questions  were  used  to  guide  the  red  team  participants  in  creating 
their  AAR  video.  All  questions  were  not  answered  in  all  videos,  but  this  was  intended  to 
ensure  that  each  video  contained  similar  information  that  would  be  relevant  to  a 
subsequent  player. 

1 .  Why  did  you  pick  your  defense  course  of  action  (COA)  (rovers  versus 
guards)? 

2.  What  was  your  planned  scheme  of  maneuver  for  defense? 

3.  Did  anything  prevent  you  from  executing  your  planned  mission? 

4.  Was  your  mission  a  success?  Why  or  why  not? 

5.  What  if  anything  would  you  do  different? 
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Similarly,  the  blue  team  videos  addressed  the  following  questions: 


1 .  Why  did  you  pick  your  release  point? 

2.  What  was  your  planned  scheme  of  maneuver? 

3.  Did  anything  prevent  you  from  getting  to  the  objective,  if  so  what? 

4.  What  happed  at  the  objective? 

5.  Was  your  mission  a  success?  Why  or  why  not? 

6.  What  if  anything  would  you  do  different? 

Participants  also  filled  out  a  short  demographic  survey  as  well  as  post  task  survey. 

Once  Group  1  completed  recording  all  20  videos,  the  researchers  reviewed  and 
ranked  the  videos  based  on  both  content  and  presentation.  This  was  an  artificial  ranking 
but  it  is  what  players  would  expect  from  an  actual  community  site  that  would  rank  videos 
based  on  viewer  feedback.  Since  this  study  did  not  have  an  actual  online  community  to 
rank  the  videos,  there  needed  to  be  a  way  to  present  the  videos  to  the  second  group  along 
with  some  feedback  about  which  were  the  best  ones.  In  practice,  players  would  expect  a 
game-based  social  network  where  users  give  “likes”  and  add  comments  to  the  best  videos 
and  the  best  videos  will  have  the  most  “views.”  The  rankings  were  important  and  they 
ensured  that  the  videos  the  second  group  viewed  were  the  most  engaging.  For  this 
experiment,  the  actual  ranking  of  AAR  videos  is  not  important  because  our  focus  was  on 
the  affect  the  videos  had  on  subsequent  players  who  viewed  the  videos. 

Once  all  the  videos  had  been  ranked,  the  top  two  from  both  the  blue  and  red  sides 
were  chosen  for  the  test  group  to  watch.  However,  further  analysis  of  the  videos  after 
ranking  revealed  that  all  of  the  red  videos  utilized  the  red  defense  of  guard  towers,  and 
none  of  the  blue  videos  utilized  the  strategy  of  starting  at  release  point  2.  Since  the  goal 
of  the  experiment  was  to  see  if  the  Group  1  videos  affected  the  behavior  of  Group  2  (the 
dependent  group),  researchers  created  two  artificial  videos  to  add  to  the  set  of  videos  for 
Group  2  to  view.  One  video  was  a  scenario  in  which  the  red  forces  would  use  a  roving 
guard  strategy.  The  other  video  was  a  scenario  in  which  the  blue  forces  would  start  out 
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from  release  point  2.  With  a  set  of  AAR  videos  covering  the  full  breadth  of  strategic 
options,  the  study  could  determine  if  the  additional  information  caused  players  in  Group 
2  to  differ  from  their  decisions  in  both  mission  planning  and  during  the  operation. 

4.  Group  2  Experiment 

Researchers  solicited  and  organized  Group  2  in  the  same  manner  as  Group  1.  In 
addition,  moderators  screened  users  to  ensure  that  no  one  who  participated  in  the  first 
group  participated  in  the  second  group.  After  the  formation  of  teams,  moderators  gave 
Group  2  the  same  opportunities  and  time  as  Group  1  to  complete  tutorials  and  a  practice 
scenario.  After  familiarization,  the  moderators  administered  the  exact  same  mission  brief 
to  Group  2  that  they  gave  to  Group  1.  After  this  brief,  the  moderators  allowed  Group  2  to 
watch  about  15  minutes  of  three  short  AAR  videos  from  Group  1  completing  and 
narrating  their  missions.  After  watching  the  videos,  moderators  gave  Group  2  the  same 
planning  time  as  Group  1  before  starting  the  mission. 

At  the  end  of  their  mission,  the  second  group  completed  a  short  survey  about  their 
experience.  The  questions  in  their  survey  included  the  following: 


1.  How  much  did  the  videos  influence  your  decisions  in  either  mission 
planning  or  during  the  operation? 

2.  How  useful  did  you  think  the  videos  were  and  would  you  recommend  that 
other  users  watch  them? 

3.  How  much  did  the  videos  contribute  to  your  overall  success  or  failure  in 
completing  the  mission? 


C.  RESULTS 

The  first  metric  collected  from  each  scenario  was  mission  success.  Blue  teams 
succeeded  in  the  mission  if  they  could  get  the  hostage  from  the  prison  complex  to  the  PZ 
for  extraction.  Red  teams  succeed  if  they  prevented  the  hostage  and  blue  forces  from 
getting  to  the  PZ.  Moderators  briefed  both  sides  that  if  either  team  harms  the  hostage,  that 
team  would  lose;  however,  this  never  happened  during  the  experiment.  The  results  of 
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Group  1  produced  four  blue  winning  sessions  and  one  red  winning  session,  and  Group  2 
produced  four  red  winning  sessions  and  one  blue  winning  session  (see  Figure  4). 


Figure  4.  Blue  versus  Red  Winners  by  Group. 


In  the  scenario,  the  blue  forces  had  to  choose  either  release  point  1,  2  or  3.  In  the 
first  group,  blue  players  chose  release  point  1  four  times  and  only  choose  release  point  3 
once.  Group  1  blue  forces  never  selected  release  point  2.  Group  2  only  selected  release 
point  1  once  and  selected  release  point  2  and  3  twice  each  (see  Figure  5).  In  discussion 
with  the  Group  I  players  after  the  session,  all  of  the  players  that  chose  release  point  1  did 
so  because  they  felt  it  offered  the  most  cover  and  concealment. 
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Figure  5.  Blue  versus  Red  Winners  by  Group. 


The  red  team  had  the  option  to  choose  one  of  two  defense  strategies.  The  two 
options  were  defending  with  two  AI  soldiers  positioned  in  guard  towers  or  two  AI 
soldiers  assigned  roving  guard  routes.  Group  1  only  chose  the  guard  tower  option  and 
Group  2  chose  the  guard  tower  option  all  but  once.  In  discussion  with  the  red  teams  that 
chose  the  guard  tower  option,  all  stated  that  the  reason  was  to  give  them  advanced  notice 
of  blue  forces  arriving  (see  Figure  6). 

Figure  6.  Red  Team  Strategy — Tower  versus  Rover. 
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After  each  mission,  moderators  gave  all  participants  post  task  surveys  to  fill  out. 
The  post  task  surveys  only  contained  three  questions  that  common  for  both  groups: 


1.  On  a  scale  of  1-5  (1  being  “bad”  and  5  being  “great”)  How  would  you  rate 
your  game  experience? 

2.  On  a  scale  of  1-5  (1  being  “not  fun”  and  5  being  “fun”)  How  would  you 
rate  how  fun  the  game  was? 

3.  Would  you  create  videos  for  other  players  to  watch?  Yes/No 


The  average  score  for  game  experience  for  Group  1  was  3.7,  and  the  average 
score  for  Group  2  was  4.1.  The  average  score  for  how  fun  the  game  was  for  Group  1  was 
3.8  and  the  average  score  for  Group  2  was  4.5  (see  Figure  7).  The  game  experience 
means  between  Group  1  and  Group  2  did  not  have  any  significant  statistically  difference 
(see  Figure  9).  However,  there  was  a  statistically  significant  difference  between  Group  1 
and  Group  2  with  regard  to  the  mean  scores  on  questions  two.  Analysis  showed  that  at  a 
95  percent  confidence  level,  the  amount  of  fun  that  Group  2  recorded  was  higher  than 
what  Group  1  had  (see  Figure  8). 
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Figure  8. 


Analysis  of  Means  for  Question  2 — Rate  Your  Game  Experience. 


Figure  9. 


J  ▼  Analysis  of  Means 


Avg  =  3.8S0 


Group 


A  Means  Comparisons 

A  ▼  Comparisons  for  each  pair  using  Student's  t 
A  Confidence  Quantile 
t  Alpha 

2,02439  0,05 

A  LSD  Threshold  Matrix 

Abs(DiO-LSD 

2  1 
2  -0.56405  -0.16405 

1  -0.16405  -0.56405 


Positive  values  show  pairs  of  means  that  are  significantly  different. 


Analysis  of  Means  for  Question  2 — Rate  How  Fun  the  Game  Was. 
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Analyis  of  Question  3  revealed  that  about  in  total  72.5  pereent  of  the  partieipants 
said  that  they  would  create  videos  for  others  to  watch.  Group  1  participants  answered  yes 
75  percent  and  Group  2  answered  yes  70  percent.  Threre  was  no  statisicical  difference  in 
the  two  groups  scores  (see  Figures  10  and  11). 

Figure  10.  Mosaic  Plot — Would  You  Create  Videos?  By  Group. 


Figure  1 1 .  Analysis  of  Means  for  Proportions — Would  You  Create  Videos? 
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Group  2’s  survey  asked  the  Group  2  participants  eight  additional  questions  that 
related  directly  to  the  videos  that  they  watched  before  playing  their  session.  The 
questions  were: 


1.  On  a  scale  of  1-5  (1  being  “not  useful”  and  5  “very  useful”)  How  would 
you  rate  how  useful  the  videos  were  to  your  experience? 

2.  Would  you  recommend  watching  user  videos  for  other  participants?  Yes/ 
No 

3.  Do  you  feel  that  the  user  videos  influenced  your  decisions  during  mission 
planning?  Yes/No 

4.  If  yes,  rate  the  influence  on  a  scale  of  1-5  (1  being  “little  influence”  and  5 
“highly  influenced”) 

5.  Do  you  feel  that  the  user  videos  influenced  your  decisions  during  the 
operation?  Yes/No 

6.  If  yes,  rate  the  influence  on  a  scale  of  1-5  (1  being  “little  influence”  and  5 
“highly  influenced”) 

7.  Do  you  feel  that  the  user  videos  contributed  to  your  success  or  failure  of 
the  operation?  Yes/No 

8.  If  yes,  rate  the  contribution  on  a  scale  of  1-5  (1  being  “little  contribution” 
and  5  “high  contribution”) 


On  a  scale  of  1-5,  Group  2  rated  the  videos  as  a  4.25  in  terms  of  helpfulness  and  100 
percent  of  the  participants  said  that  they  would  recommend  to  other  players  to  watch  the 
videos  (see  Figure  12). 
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Figure  12.  Usefulness  of  Videos  and  Would  You  Reeommend  Them. 


Usefulness  and  Recommendations 


“very  usefur)  Hov^wouW you  rate  how  useful 
the  videos  wereto  your  experience? 


Would  you  recommend  Vi^atching  user 
videosfor  other  participants? 


Ninety-five  percent  of  Group  2  participants  said  that  the  videos  affected  their 
mission  planning  and  rated  the  effect  on  mission  planning  as  3.74  (see  Figure  13). 


Figure  13.  Effects  of  Videos  on  Mission  Planning. 


Do  you  feel  that  the  user  videos  influerKed  If  yes,  rate  the  influence  on  a  scale  of  1-5 

yourdecisbnsduring  mtssbn  planning?  (1  being  "littleinfluence"  and  5  "highly 

influenced") 


25 


In  contrast,  only  55  percent  of  Group  2  participants  said  that  the  videos  affected 
their  decision  during  the  mission.  Those  who  said  it  did  rated  the  effects  as  3.74  out  of  5. 
(see  Figure  14).  The  effects  on  mission  planning  versus  decisions  during  the  operation 
showed  that  the  videos  affected  Group  2  participants  during  mission  planning  40  percent 
more  than  they  affected  decision  during  the  operation.  The  seore  with  regard  to  how 
much  the  video  affected  both  were  very  similar  (see  Figure  15). 


Figure  14.  Effects  of  Videos  during  the  Operation. 


Do  you  feel  that  the  user  videos  influenced 
yo  u  r  d  eci  sb  ns  d  uring  th  e  o  petatkxi  ? 


If  yes,  rate  the  influence  on  a  scale  of  1-5 
(1  being  "little influence"  and  5  "highly 
influenced”) 
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Figure  15.  Effects  of  Videos  during  Mission  Planning  versus  the  Operation. 


D.  DISCUSSION 

I.  Artificial  Online  Community 

Creating  an  actual  online  community  of  ESP  game  players  to  conduct  this  study 
was  outside  of  this  study’s  scope  and  resources.  Therefore,  Group  I’s  primary  purpose  in 
this  study  was  to  provide  the  video  narration  for  the  researchers  to  use  to  create  an 
artificial  online  community.  This  artificial  online  community  would  be  the  environment 
for  the  second  group  to  play  in.  In  a  real  online  community,  users  would  rank  and  score 
videos  based  on  the  their  content  and  usefulness,  and  it  would  be  easy  for  the  other  users 
to  find  to  the  best  and  most  watched  videos.  Given  the  time  constraint  for  the  experiment. 
Group  2  did  not  have  the  necessary  time  to  watch  all  ten  full-length  videos  created  by 
Group  1.  Therefore,  the  researchers  needed  to  choose  the  best  videos  and  then  edit  each 
video  selected  from  roughly  15  minutes  to  3-5  minutes.  This  was  done  by  speeding  up 
the  playback  of  the  video  in  between  the  voice  narration  parts  of  the  video. 

However,  Group  1  had  a  secondary  role  in  this  experiment.  Because  they  did  not 
receive  any  AAR  preview  prior  to  executing  the  missions,  their  strategic  and  tactical 
decisions  were  used  as  a  baseline  for  how  players  might  approach  the  scenario  without 
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any  opportunity  to  see  how  others  attempted  it  before  them.  See  Section  5  for  further 
discussion  on  this  point. 

Another  issue  that  creating  an  artificial  online  community  presented  was  the  lack 
of  variety  in  videos  created  by  Group  1 .  Group  1  conducted  five  separate  sessions  in  total 
and  in  those  five  sessions,  the  Group  1  blue  team  selected  release  point  1  for  its  starting 
point  4  times  and  release  point  3  only  once.  Not  one  Group  1  team  selected  release  point 
2  as  a  starting  position.  The  Group  1  red  teams  followed  suit  and  selected  the  tower 
defense  strategy  over  the  roving  guard  strategy  each  time  as  well.  After  the  Group  1 
session,  the  selection  of  videos  chosen  for  Group  2  to  watch  did  not  have  any  release 
point  1  or  roving  guard  strategies  examples  for  Group  2  to  watch.  In  order  to  present 
Group  2  with  a  variety  of  videos,  two  artificial  videos  were  made,  one  to  illustrate  an 
example  of  release  point  2  and  one  to  illustrate  an  example  of  defending  with  roving 
guards.  Due  to  the  small  sample  size  for  video  selection  coupled  with  the  fact  that  the 
primary  focus  for  Group  2  was  how  they  would  react  to  the  videos,  adding  two  artificial 
videos  was  a  change  that  the  researchers  felt  was  warranted. 

One  outcome  from  presenting  Group  2  with  a  variety  of  videos  that  showed 
multiple  ways  to  complete  the  mission  was  the  Group  2  blue  forces  and  red  forces 
showed  more  variability  in  their  strategy  selection  than  Group  1 .  However,  due  to  such  a 
small  sample  size  of  videos  and  sessions,  this  data  cannot  support  any  conclusions 
without  additional  research. 

2.  Winning  Percentage — Group  1  versus  Group  2 

The  blue  team  would  win  the  scenario  if  it  successfully  moved  the  hostage  to  the 
PZ  before  the  red  team’s  QRF  arrived.  For  the  red  team,  preventing  the  blue  forces  from 
reaching  the  PZ  with  the  hostage  and  ensuring  the  safety  of  the  hostage  constituted  a 
successful  mission.  Group  1  produced  four  out  of  five  blue  winners.  Group  I’s  blue  team 
used  release  point  1  three  times  successfully  and  release  point  3  once  successfully.  In  the 
second  group  of  trials,  the  blue  force  was  only  successful  once.  The  lone  successful  run 
from  blue  forces  in  the  second  group  originated  at  release  point  2  and  was  against  a  red 
force  utilizing  the  towers.  In  the  other  four  sessions  in  Group  2,  the  Group  2  red  teams 
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won  each  time.  The  major  difference  between  the  red  forces  in  Group  1  and  Group  2  was 
the  use  of  a  heavy  machine  gun  inside  the  prison.  This  was  a  strategy  that  was 
highlighted  in  one  of  the  videos  and  all  of  the  red  teams  in  Group  2  adapted  to  this 
strategy.  The  adoption  of  this  strategy  was  primary  responsible  for  the  shift  in  red’s 
winning  percentage  from  20  percent  to  80  percent,  thus  providing  further  evidence  that 
the  AARs  did  affect  subsequent  player  strategy  and  behavior. 

Blue  forces  may  have  redistributed  their  starting  points  due  to  watching  the 
videos,  changing  from  starting  at  release  point  1  four  times  and  release  point  2  once,  to 
starting  at  release  point  1  once,  release  point  2  and  3  twice  each.  Anecdotal  evidence 
revealed  that  most  Group  1  blue  forces  chose  release  point  1  because  they  felt  it  offered 
the  most  cover  and  concealment.  Group  2’s  blue  forces,  however  were  able  to  watch  two 
successful  videos  in  which  the  cover  and  concealment  afforded  to  the  blue  forces  from 
release  point  1  did  not  outweigh  the  easier  access  to  the  objective  that  release  point  2  and 
release  point  3  offered.  In  fact,  even  though  the  blue  forces  were  less  successful  in  the 
second  group,  it  was  not  due  to  the  release  point  that  they  selected.  In  Group  2,blue 
forces  reached  the  prison  complex  in  all  of  the  scenarios  with  more  combat  power  than 
they  did  in  Group  1 .  As  stated  earlier,  the  biggest  reason  for  the  shift  was  due  to  Group 
2’s  red  forces  use  of  a  heavy  machine  gun  at  the  objective.  Group  2’s  blue  forces  also 
utilized  successful  examples  they  viewed  from  Group  I’s  videos  such  as  use  of  an  AT4 
rocket  launcher  on  the  guard  towers  and  utilization  of  easier  access  points  in  and  out  of 
the  objective. 


3.  Increase  in  “Fun”  Level  for  Group  2 

As  stated  earlier.  Group  2  reported  a  statistically  significant  difference  in  the  level 
of  fun  that  it  had  compared  to  what  Group  1  reported.  The  data  collected  from  this 
experiment  does  not  point  to  any  underlying  reasons  for  this  shift;  however,  future 
experiments  should  examine  a  few  hypotheses.  First,  it  may  be  that  playing  the  game 
after  you  have  seen  others  play  creates  more  of  a  competitive  game  environment.  Second, 
it  could  be  that  after  the  Group  2  players  saw  their  peers  from  Group  1  complete  the 
mission  successfully  and  report  how  they  did  it,  it  gave  more  incentive  for  Group  2 
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players  to  not  only  complete  the  mission  as  well,  but  do  even  better  than  the  previous 
group.  Another  reason  may  be  that  watching  videos  gave  Group  2  more  context  about  the 
mission  and  highlighted  more  options  that  Group  1  was  not  able  to  fully  see.  This  added 
information  and  context  may  have  made  the  game  more  fun  for  Group  2.  Finally,  the 
reason  for  the  difference  in  the  fun  levels  between  the  two  groups  could  be  that  creating 
the  videos  was  less  fun  that  watching  them.  Since  Group  1  had  to  create  the  videos,  they 
recorded  lower  fun  levels  than  the  group  who  did  not  have  to  create  them.  Although  it  is 
unlikely  that  the  video  creation  decreased  Group  I’s  fun  level  significantly  since  both 
Group  1  and  Group  2  recorded  very  similar  game  experience  scores,  the  effects  of 
creating  videos  on  the  fun  level  cannot  be  ruled  out. 

4.  Usefulness  of  Videos 

Group  2  unanimously  responded  that  it  would  recommend  watching  the  videos  to 
other  participants.  This  number  is  hard  to  ignore  and  evidence  shows  both  qualitatively 
and  quantitatively  that  watching  the  videos  made  Group  2  better  at  completing  the 
mission  than  Group  1.  Compare  this  number  with  the  fact  that  95percent  of  Group  2’s 
players  said  the  video  affected  mission  planning;  one  can  easily  infer  that  Group  2  feels 
that  other  users  should  use  these  videos  for  their  mission  planning  as  well. 

Furthermore,  if  the  AAR  videos  were  useful  to  Group  2  players  in  exploring 
options  and  making  decisions  about  how  to  execute  the  mission,  then  it  also  suggests  that 
an  analyst  (e.g.,  acquisition,  operations)  would  also  be  able  to  extract  the  same  useful 
information  from  the  AAR  videos.  This  implies  that  when  an  analyst  designs  an  ESP 
scenario  to  answer  a  specific  question,  he  must  ensure  that  the  scenario  forces  the  game 
player  to  solve  a  problem  exactly  on  that  point  so  that  it  will  be  addressed  in  the  resulting 
AAR  video. 

5.  Effects  on  Mission  Planning  Decisions  versus  Operational  Decisions 

Ninety-five  percent  of  Group  2’s  players  responded  that  the  videos  affected  their 

mission  planning  while  only  55percent  said  it  affected  their  actual  decision  during  the 

game.  The  difference  in  these  proportions  suggests  that  while  the  Group  2  players  used 

other  players’  experience  for  planning,  once  the  mission  began,  what  they  saw  and 
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learned  was  not  as  valuable  as  the  information  gathered  as  the  mission  unfolded.  This  is 
understandable  since  the  mission  they  completed  was  against  other  human  players  and 
therefore,  the  videos  could  not  predict  exactly  what  the  Group  2  players  would  face.  It 
also  suggests  that  during  mission  planning,  it  is  easier  to  consider  other  players’  opinions, 
but  once  the  game  begins,  the  players  are  now  in  reactionary  mode,  without  time  to 
consider  the  videos.  It  may  also  suggest  that  the  AARs  were  especially  effective  in 
planning  such  that  operational  decisions  went  generally  according  to  expectations  thereby 
requiring  little  input  from  the  AARs.  In  either  case,  we  are  encouraged  that  the  AARs 
were  indeed  useful  and  did  provide  valuable  assistance. 

6.  Research  Questions 

The  experiment  in  this  thesis  answered  three  of  the  four  research  questions 
proposed  with  quantifiable  data.  Researchers  relied  on  qualitative  evidence  collected 
from  the  Group  1  videos  to  answer  the  one  question  not  supported  by  quantifiable  data 
collected  from  surveys  or  game  play. 

a.  Is  It  Reasonable  to  Expect  ESP  Game  Players  to  Create  and  Narrate 
Game  Play  Video  AARs? 

Survey  data  suggest  that  a  large  majority  of  players  who  participated  in  the  study 
would  create  and  narrate  game  play  videos.  Overall  72.5  percent  of  the  participants  said 
that  they  would.  Although  the  sample  size  is  too  small  to  reasonably  predict  how  many 
ESP  players  would  actually  produce  videos,  the  results  suggest  that  a  good  portion 
would. 


b.  Does  Availability  of  Game  Play  Video  AARs  Increase  User  Feedback 
and  Participation  in  ESP  Game  Play? 

The  study  did  not  directly  reveal  whether  the  use  of  game  play  video  AARs  would 
increase  participation.  However,  results  did  show  that  the  players  who  watched  videos 
had  more  fun  than  those  that  did  not.  This  data  coupled  with  the  results  from  the  study  at 
Fort  Bliss  where  soldiers  indicated  that  the  fun  level  was  the  most  important  factor  for 
their  participation  in  an  ESP  game  (Vogt  et  al.  2015).  If  this  data  holds  true,  then  one  can 

infer  that  the  use  of  videos  would  increase  participation. 
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The  videos  also  increased  the  feedback  from  the  soldiers  by  providing  the  “why” 
in  their  decision.  Most  of  what  was  collected  from  the  players  during  their  ESP  game 
session  was  data  that  answered  “what.”  The  game  was  able  to  determine  what  tactical 
decision  and  force  structures  the  players  made,  but  did  not  tell  why  they  made  them.  The 
only  way  to  get  the  answer  as  to  why  the  players  made  certain  decision  was  to  watch  the 
AAR  videos.  The  videos  alone  provided  the  context  for  the  various  decisions  that  players 
made  during  the  game. 

c.  Will  Game  Play  Video  AARs  Provide  Any  Useful  Feedback  to 
Developers? 

The  main  purposed  of  the  videos  is  to  capture  the  context  around  certain  decisions 
made  rather  than  act  as  a  mechanism  to  capture  large  amounts  of  metrics  for  developers. 
It  would  be  very  difficult  for  developers  gather  data  automatically  from  the  videos. 
However,  it  may  be  possible  for  developers  dive  into  the  most  watched  or  best-rated 
videos  or  videos  from  the  users  with  the  highest  score  to  learn  why  they  are  making 
certain  decisions.  As  mentioned  earlier,  if  scenario  designers  create  problems  in  their 
scenarios  that  directly  address  the  questions  they  want  players  to  answer,  then  the  AARs 
will  contain  information  about  solving  that  problem  that  are  useful  to  the  developer. 
Guidelines  for  how  to  design  scenarios  to  answer  a  specific  question  could  be  a  fruitful 
topic  of  further  research.  This  thesis  shows  that  AARs  would  be  an  effective  way  to 
capture  the  necessary  data  that  answers  the  question. 

d.  Will  Game  Play  Video  AARs  Provide  Any  Useful  Information  to  Other 
Players  and  Enhance  Their  Game  Play? 

The  information  present  in  the  videos  did  provide  useful  information  for  the  other 
players  and  made  the  game  more  fun  for  the  players  that  watched  the  videos.  Every 
player  that  watched  videos  said  that  they  would  recommend  other  users  watch  the  videos 
as  well.  In  addition,  of  the  participants  that  watched  videos,  95  percent  said  that  the 
videos  affected  their  mission  planning  and  55  percent  said  it  affected  their  in  game 
decisions.  Additionally,  players  that  watched  the  videos  recorded  a  statistically  higher 
level  of  fun  than  the  players  that  played  without  watching  the  videos  beforehand. 
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IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


Although  this  study  had  limitations,  it  was  an  overall  success.  It  produced  a 
number  of  takeaways  that  should  be  of  use  to  the  ESP  community.  The  limitations  in  this 
study  could  be  easily  overcame  or  mitigated  in  future  studies,  especially  if  future  studies 
increase  their  scope  and  resources. 

A.  LIMITATIONS 

This  study  had  three  main  limitations  that  future  studies  should  seek  to  overcome 
or  mitigate.  They  were  the  lack  of  an  actual  ESP  type  game,  a  small  sample  size,  and  the 
lack  of  an  actual  online  gaming  community. 

I.  VBS3 

While  VBS3  was  useful  for  this  study,  it  has  its  limitations.  Eirst,  VBS3  is  a 
simulation  training  software  designed  to  train  soldiers  in  collective  tasks  at  the  platoon 
and  squad  level.  VBS3  is  not  a  feasible  solution  for  an  ESP  game  because  the  models  in 
VBS3  are  not  easily  changeable.  An  ESP  game  will  need  to  offer  the  players  a  system 
with  changeable  design  points  so  that  researchers  can  evaluate  the  use  of  the  system  from 
various  configurations  as  well  as  its  use  in  various  ways.  VBS3  may  have  some  limited 
capability  to  do  some  configuration  changes,  but  given  the  resources  available  for  this 
study,  we  were  not  able  to  utilize  VBS3  in  that  manner.  In  fact,  the  game  only  presented 
organizational  and  tactical  options  to  the  players. 

Another  limitation  in  VBS3  is  the  functionality  of  the  AT  The  scenario  from  the 
experiment  required  game  AI  to  control  some  of  the  forces.  Although  the  researchers 
tested  the  game  scenario  through  many  rehearsals  and  trial  runs  with  the  game’s  AI,  it 
was  impossible  to  present  the  AI  with  every  situation  possible.  Throughout  both  groups, 
situations  arose  where  the  AI  did  not  act  in  the  manner  that  the  players  would  expect.  Eor 
this  reason,  the  players  were  informed  which  team  members  were  AI  and  which  where 
human-in-the-loop.  This  allowed  moderators  to  manage  the  players’  expectations  since 
the  players  did  not  expect  the  AI  to  be  as  responsive  as  human  teammates. 
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2.  Sample  Size  and  Participant  Demographics 

The  amount  of  time  available  for  this  study  coupled  with  resources  available  did 
not  allow  for  a  large  sample  size.  Although  40  individuals  participated  in  the  experiment, 
only  five  sessions  were  run  in  each  group.  This  gave  a  large  sample  size  of  participants, 
but  a  small  sample  size  of  sessions  and  videos  to  present  to  the  second  group.  Another 
limitation  in  the  sample  size  was  the  number  of  “gamers”  in  each  group.  While  the 
moderators  sought  to  balance  the  teams  by  how  much  time  each  participant  regularly 
played  video  games,  ideally  the  target  audience  for  this  type  of  test  should  be  individuals 
who  regularly  play  video  games.  When  the  Army  fully  develops  ESP,  soldiers  will  most 
likely  play  it  on  a  voluntary  basis  (Vogt  et  al.  2015).  For  this  reason,  future  studies  need 
to  measure  the  effects  of  videos  on  ESP  players  who  are  likely  to  play  ESP  games. 

3.  Artificiality  of  the  ESP  Community 

The  main  effort  of  ESP  will  be  the  creation  and  management  of  an  online 
community  for  players  to  collaborate  and  developers  to  harvest  desired  metrics  (Vogt, 
2016).  In  this  environment,  players  will  most  likely  create  and  share  videos.  Resources 
did  not  allow  for  the  actual  creation  of  such  an  environment.  In  addition,  in  an  ESP 
environment,  players  who  make  and  share  videos  will  most  likely  create  them  on  their 
own.  In  this  study,  moderators  had  to  walk  players  through  the  creation  of  the  videos  and 
then  edit  the  videos  down  so  that  the  second  group  could  watch  a  variety  of  videos  in  the 
time  allotted. 

B.  RECOMMENDATIONS 

As  stated  earlier,  this  pilot  study  provides  a  baseline  for  future  studies. 

Additionally,  it  is  recommended  that  future  ESP  games  and  communities  be  designed  to 

allow  ESP  game  players  to  create  and  share  game  play  videos  as  well  as  communicate  via 

chat  room  and  message  boards  to  exchange  ideas.  While  the  underlying  reasons  for  the 

increase  in  fun  from  the  videos  are  still  unknown,  the  data  supports  the  assumption  that 

when  players  play  after  watching  the  videos,  they  have  more  fun.  Since  an  ESP  game  will 

already  need  a  collaborative  environment  for  game  players  to  share  information  with 

capability  and  material  developers  anyway,  ESP  developers  should  also  include  an 
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environment  where  players  can  communicate  with  one  another  as  well.  This  would  not 
only  allow  players  to  solicit  ideas  from  each  other,  but  also  allow  developers  to 
investigate  why  players  are  making  the  design  or  usage  decisions  they  are  making. 
Additionally,  future  studies  should  seek  to  determine  exactly  what  the  underlying  reasons 
for  the  increase  in  fun  from  the  game  play  videos  was.  Two  reasons  that  should  be 
explored  are  (1)  game  play  videos  provide  more  context,  and  (2)  game  play  videos  create 
higher  completion  rates. 

C.  CONCLUSIONS 

Although  this  pilot  study  had  some  limitations,  it  provided  four  key  takeaways  for 
future  ESP  studies  and  ESP  developers.  Eirst,  results  suggest  that  incorporating  user 
videos  into  an  ESP  community  would  increase  participation  as  well  as  increase  the  fun  of 
the  game.  Second,  the  data  from  this  experiment  indicates  that  the  videos  did  in  fact 
affect  the  players’  behaviors.  Third,  every  player  who  watched  the  videos  recommended 
that  other  users  watch  the  videos  as  well.  Einally,  over  70  percent  of  the  participants 
responded  that  they  would  create  and  narrate  videos  in  an  ESP  game  environment. 

The  four  key  findings  from  the  experiment  together  answer  three  of  the  four 
research  questions  this  thesis  set  out  to  answer.  The  lone  question  that  did  not  have  any 
quantifiable  data  was  did  the  videos  provide  any  useful  feedback  for  developers?  This 
question  was  harder  to  qualify  because  the  game  used  in  this  study  was  not  an  actual  ESP 
game,  and  the  players  who  created  the  videos  followed  a  script  to  ensure  that  all  of  the 
relevant  parts  of  the  mission  were  covered.  However,  the  data  that  the  users  presented  did 
provide  context  for  the  decisions  that  the  players  made.  Therefore,  the  videos  would  not 
provide  the  “what”  to  the  developers;  instead,  they  would  provide  the  “why”  to  them. 
This  would  allow  developers  to  understand  the  logic  behind  why  players  made  decisions. 

Although  this  was  a  pilot  study,  the  results  suggest  that  there  may  be  some 
positive  outcomes  to  incorporating  a  video  sharing  function  into  the  ESP  game 
community.  In  a  previous  study,  researchers  found  that  the  number  one  reason  soldiers 
would  participate  in  an  ESP  game  during  their  time  off  was  fun  (Vogt  et  ah,  2015). 
Therefore,  if  Soldiers  are  playing  in  an  environment  where  they  can  watch  videos,  this 
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should  increase  the  likelihood  that  they  will  participate.  However,  the  ESP  community 
will  need  to  monitor  the  videos  and  community,  both  to  ensure  to  the  quality  of  the 
videos  presented  as  well  as  to  prevent  groupthink.  If  players  all  flock  to  one  way  of 
thinking,  it  may  diminish  the  number  of  viable  prototypes  tested.  This  was  evident  in  the 
Group  2’s  red  forces  use  of  a  heavy  machine  gun  to  guard  the  prison.  Once  Group  2’s  red 
forces  learned  through  the  view  of  the  videos  that  this  option  was  highly  successful, 
future  users  naturally  chose  it  as  an  option  as  well. 

D.  FUTURE  WORK 

This  study  only  scratched  the  surface  of  the  effects  of  videos  on  game  play  in  an 
ESP  game  and  researchers  need  to  conduct  future  studies  before  the  data  from  this  study 
can  assert  anything  finite.  In  the  future,  studies  need  to  increase  the  participant  pool  by 
using  an  online  game  that  allows  researchers  to  solicit  participants  remotely  and  allow 
participants  to  participate  on  their  own  time.  This  may  require  a  less  controlled 
experiment,  but  should  vastly  increase  the  number  of  data  points  for  analysis.  This  also 
allows  the  study  to  get  more  “gamers”  involved.  Additionally,  any  follow  on  study 
should  also  have  a  real  online  game  community  where  players  can  share  their  own  videos 
without  the  use  of  researchers  picking  the  top  videos.  This  would  eliminate  the  limitation 
of  the  artificial  community  and  should  increase  accuracy. 
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APPENDIX  A.  BLUE  OPERATIONS  BRIBE 


Blue  Brief; 

You  are  part  of  a  4  man  rescue  team  that  is  stationed  in  the  country  of  Ostil,  right  next  to  the  Gorgas 
border. 

Intel  has  just  learned  that  Nicole  Lewis,  reporter  for  news  7,  has  been  taken  as  a  political  prisoner  in  the 
country  of  Gorgas,  by  regular  Gorgarian  Army  soldiers. 

Lewis  is  being  held  captive  in  an  abandoned  prison  complex  near  the  Gorgarian/Ostilian  border. 

Currently,  it  is  believed  that  she  is  only  being  held  captive  by  a  very  small  (2-4  soldiers)  force.  However, 
within  the  next  few  hours,  she  is  expected  to  be  moved  to  the  Gorgarian  capital  city  of  Masra. 

The  prison  complex  security  team  is  believed  to  be  2-4  soldiers  with  the  capability  of  a  QRF  force  to 
arrive  in  8  minutes  if  they  are  called. 

Prison  guards  at  the  complex  have  good  communications  with  the  QRF,  therefore,  QRF  will  most  likely 
be  called  once  your  element  is  spotted  or  you  engage  the  QRF. 

Your  mission  is  to  conduct  a  rescue  mission  at  Qbjective  Snyder  (Prison  Complex)  where  you  will  need  to 
secure  the  prisoner  and  move  to  PZ  nelson  for  extraction. 


37 


Task  Org 


•  TF  Striker 

-TF  Eagle 

•  1  UH60L-  M134 

-TF  Wildcat 

•  4  US  Soldiers. 

•  1  Weapons  and  Ammo  create  at  RP. 


Situation 


•  Nicole  Lewis,  reporter  for  news  7,  has  been  taken  as  a  political 
prisoner  in  the  country  of  Gorgas,  by  regular  Gorgarian  Army 
soldiers. 

•  Lewis  is  being  held  captive  in  an  abandoned  prison  complex  near 
the  Gorgarian/Ostilian  border. 

•  Currently,  it  is  believed  that  she  is  only  being  held  captive  by  a  very 
small  (2-4  soldiers)  force.  However,  within  the  next  few  hours,  she 
is  expected  to  be  moved  to  the  Gorgarian  capital  city  of  Masna. 

•  Prison  complex  security  team  is  believed  to  be  2-4  soldiers  with  the 
capability  of  a  QRF  force  to  arrive  in  8  minutes  if  they  are  called. 

•  Prison  guards  at  the  complex  have  good  communications  with  the 
QRF,  therfore,  QRF  will  most  likley  be  called  once  your  element  is 
spotted  or  you  engage  the  QRF. 
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Mission 

•  Your  mission  is  to  conduct  a  rescue  mission  at 
Objective  Snyder  (Prison  Complex)  where  you 
will  need  to  secure  the  prisoner  and  move  to 
PZ  nelson  for  extraction. 


Coordinating  Instruction/Support 

•  Select  a  release  point  for  your  insertion  point. 

•  Ammo  and  weapons  create  available  at  your 
RP  to  acquire  any  additional  weaponsyou  may 
want. 
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Nicole  Lewis,  reporter  for  news  7. 
Being  held  captive  at  prison  complex. 


Possible  Open  Gate 
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RPl  -  16SEE4720459695 
RP2  -  16SEE4771659503 
RP3  -  16SEE4767559963 
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APPENDIX  B.  RED  OPERATIONS  BRIEE 


Red  Brief: 

You  are  from  the  country  Gorgas  and  are  a  member  of  the  Gorgas  military  forces. 

You  have  captured  a  high  value  target.  It  is  an  American  News  Reporter,  Elizabeth  Brown. 

Your  country  believes  this  woman  to  be  an  American  Spy  and  you  have  been  tasked  with  providing 
security  for  the  prisoner. 

You  are  currently  holding  this  prisoner  in  an  abandoned  prison  complex  near  the  border  of  Gorgas  and 
Ostil,  which  is  where  you  pick  up  and  captured  the  American  journalist. 

Your  country  is  not  friendly  with  the  United  States,  however  you  neighbor  Ostil  is. 

Currently,  there  are  very  few  US  forces  in  the  Gorgas/Ostilian  Boarder  region,  however,  a  small  2-4  man 
rescue  attempt  may  be  likely. 

You  are  part  of  a  4  man  security  team  tasked  with  providing  security  until  the  prisoner  can  be  moved  to 
a  more  secure  location  within  the  next  few  hours 

As  the  prison  security  force  team  lead,  you  will  have  2  soldiers  that  can  either  be  placed  as  roving  guards 
or  in  the  towers.  The  other  2  soldiers  (you  and  your  battle  buddy)  can  defend  from  anywhere  on  the 
complex. 

A  QRF  force  is  available  in  the  event  that  US  forces  are  either  spotted  or  engage  your  security  force, 
however,  it  will  take  10  minutes  for  them  to  arrive,  so  you  will  need  to  ensure  that  the  prisoner  is  secure 
until  their  arrival. 

This  prisoner  is  very  important  to  the  country  of  Gorgas,  therefore,  you  must  ensure  that  her  safety  is 
maintained  throughout  the  mission.  If  she  dies  during  the  mission,  the  red  will  lose. 
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Task  Org 


•  TF  Red 

-TF  QRF  (10  minute  delay  from  notification) 

•  2  UAZ-469  with  DShK 

•  1  GAZ  Vodnik  (with  4  dismounts,  commander,  and 
driver) 

-  TF  Folsom 

•  4  OPFOR  soldiers 

•  1  Weapons  and  Ammo  create  at  RP. 


Situation 


•  Nicole  Lewis,  reporter  for  news  7,  has  been  taken  as  a  prisoner  in  the  country  of 
Gorges,  by  regular  Gorgarian  Army  soldiers. 

•  You  are  assigned  as  a  prison  guard  where  Lewis  is  being  held  captive  in  an 
abandoned  prison  complex  near  the  Gorge rian/Ostilian  boarder. 

•  You  are  part  of  a  4  man  security  team  tasked  with  providingsecurity  until  the 
prisoner  can  be  moved  to  a  more  secure  location  within  the  next  few  hours 

•  Currently,  there  are  very  little  US  forces  in  the  Gorgas/Ostilian  Boarder  region, 
however,  a  small  4  man  rescue  attempt  may  be  likely. 

•  As  the  prison  security  force  team  lead,  you  will  have  2  soldiers  that  can  either  be 
place  as  roving  guards  or  in  the  towers.  The  other  2  soldiers  (you  and  your  battle 
buddy)  can  defend  from  anywhere  on  the  complex. 

•  A  QRF  force  is  available  in  the  event  that  US  forces  are  either  spotted  or  engage 
your  security  force,  however,  it  will  take  10  minutes  for  them  to  arrive,  so  you  will 
need  to  ensure  that  the  prisoner  is  secure  unti  their  arrival. 

•  This  prisoner  is  very  important  to  the  country  of  Gorges  and  therefore,  you  must 
ensure  that  her  safety  is  maintained throughoutthe  mission.  If  she  dies  duringthe 
mission,  the  red  will  lose. 


44 


Mission 


•  Choose  defense  strategy  (either  roving  guards 
or  guard  towers). 

•  Secure  prison  complex  and  delay  any  blue 
force  attack  until  QRF  can  arrive. 

•  Ensure  prisoner's  safety  during  the  mission. 
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Roving  Guard  Route  1 


Defense  COA#2-  Roving  Guards 
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APPENDIX  C.  EXPERIMENT  SESSION  LAB  SETUP 


Server 


Lab  A 

Room  212B 
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APPENDIX  D.  DEMOGRAPHIC  SURVEY 


"Extracting  Useful  Design  Information  from  Gameplay  Videos  in  Early  Synthetic  Prototyping" 

Demographic  Survey 

Subject  #: _  Date: _ 

1.  Age: _ 

2.  Gender:  Male  Female 

3.  Are  you  currently  serving  in  the  Armed  Forces:  Yes  No 

a.  Branch: _ 

b.  Years  of  Service: _ 

c.  Flighest  Rank: _ 

4.  How  many  game  consoles  to  you  own? _ 

5.  How  many  hours  per  week  do  you  play  video  games? _ 
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APPENDIX  E.  POST  TASK  SURVEY 


"Extracting  Useful  Design  Information  from  Gameplay  Videos  in  Early  Synthetic  Prototyping" 

Post  Task  Survey 


Subject  U: _  Date: _ 

1.  One  a  scale  of  1*5  (1  being  "bad"  and  5  being  "great")  How  would  you  rate  you  game 

experience? _ 

2.  One  a  scale  of  1-5  (1  being  "not  fun"  and  5  being  "fun")  How  would  you  rate  how  fun  the  game 

was? _ 

3.  One  a  scale  of  1-5  (1  being  "not  difficult"  and  5  "very  difficult")  How  would  you  rate  difficult  the 

videos  were  to  create? _ 

4.  If  video  and  narration  creation  was  streamlined,  would  you  create  videos  for  other  players  to 
watch?  Yes/No 
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"Extracting  Useful  Design  Information  from  Gameplay  Videos  in  Early  Synthetic  Prototyping" 

Post  Task  Survey 

Subject  _  Date: _ 

1.  One  a  scale  of  1*5  (1  being  "bad"  and  5  being  "great")  How  would  you  rate  you  game 

experience? _ 

2.  One  a  scale  of  1-5  (1  being  "not  fun"  and  5  being  "fun")  How  would  you  rate  how  fun  the  game 

was? _ 

3.  One  a  scale  of  1-5  (1  being  "not  useful"  and  5  "very  useful")  How  would  you  rate  how  useful  the 

videos  were  to  your  experience? _ 

4.  Would  you  recommend  watching  user  videos  for  other  participants?  Yes/No 

5.  Do  you  feel  that  the  user  videos  influenced  your  decisions  during  mission  planning?  Yes/No 

If  yes,  rate  the  influence  on  a  scale  of  1-5  (1  being  "little  Influence"  and  5  "highly 
influenced") _ 

6.  Do  you  feel  that  the  user  videos  influenced  your  decisions  during  the  operation?  Yes/No 

If  yes,  rate  the  influence  on  a  scale  of  1-5  (1  being  "little  influence"  and  S  "highly 
influenced") _ 

7.  Do  you  feel  that  the  user  videos  contributed  to  your  success  of  failure  of  the  operation?  Yes/No 

If  yes,  rate  the  contribution  on  a  scale  of  1-5  (1  being  "little  contribution"  and  5  "high 
contribution") _ 

8.  Would  you  create  videos  for  other  players  to  watch?  Yes/No 
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ton  Red  Plan  Oroup  BlueRP  Subjects  W/l  Age  US/10  Sex  Active  Mii  Branch  Years  Serv>^  Grade  Game  Consoles  Hr>/Week  Gaming  Q1  Q2  Q3-1  Q3-2  Q4  QS  QSa  QS  Q6<  Q7  Q7a 

Runll  Tower  1  1  1  W  3S  US  M  Y  Army  16  05  3  2  4  3  4 


APPENDIX  F.  EXPERIMENT  RAW  DATA 
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6roup2Totd$  34.8  11.8  1.7  4.5  4.1  4.5  4.3  1  0.95  3.7  aS5  3.7  0.8  3.8  0.70 

Totds  34.6  11.2  1.2  3.4  3.9  4.2  0.725 
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